Skip to content

[PostgreSQL] Add a PostgreSQL wire protocol service for the table model - #19350

Draft
chrisdutz wants to merge 7 commits into
masterfrom
feature/postgresql-wire-protocol
Draft

chrisdutz wants to merge 7 commits into
masterfrom
feature/postgresql-wire-protocol

Conversation

@chrisdutz

@chrisdutz chrisdutz commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

This draft adds a PostgreSQL wire protocol service to IoTDB. PostgreSQL clients can then
query the table model, read-only: psql, JDBC tools (pgJDBC, DBeaver), Python (psycopg),
.NET (Npgsql), and BI tools that connect through PostgreSQL connectors, such as Microsoft Power BI
(Import and DirectQuery) and Excel.

It was developed in the ToddySoft DB fork of IoTDB and ported to master for this PR. The
community has asked about a PR, so I'm opening it as a draft, for two reasons:

  1. to discuss the design, and the places where this version deliberately differs from the fork
    (see "Differences from the fork" below), and
  2. to ask for help reviewing the golden files of the SQL translator (see "Help wanted").

Example

# conf/iotdb-system.properties
enable_postgresql_service=true
postgresql_service_address=127.0.0.1
postgresql_service_port=5432

Recorded with psql 17 against a standalone server from this branch, bound to 0.0.0.0 (so
TLS is required) with key_store_path set (host name and port replaced by placeholders):

$ psql "host=db.example port=5432 user=root dbname=iotdb sslmode=require" -e
iotdb=> \dn
 List of schemas
 Name  | Owner
-------+-------
 plant | root
(1 row)

iotdb=> SELECT device, date_trunc('hour', time) AS hour, avg(temp)
iotdb->   FROM plant.sensors GROUP BY 1, 2 ORDER BY 1, 2;
 device |          hour          | avg
--------+------------------------+-----
 d1     | 2026-01-15 09:00:00+00 |  21
 d2     | 2026-01-15 09:00:00+00 |  19
(2 rows)

iotdb=> INSERT INTO plant.sensors (time, device, temp) VALUES (now(), 'd3', 1.0);
ERROR:  cannot execute INSERT in a read-only transaction

$ psql "host=db.example port=5432 user=root dbname=iotdb sslmode=disable"
psql: error: connection to server at "db.example", port 5432 failed:
FATAL:  encryption required: connect with SSL (sslmode=require)

Every IoTDB database appears as a PostgreSQL schema (plant.sensors is table sensors of
database plant). Queries are translated into table-model SQL and executed by the normal engine,
so filters and aggregations stay in the engine. A construct without an exact translation is
refused with SQLSTATE 0A000 and named; it is never answered approximately.

Tree-model data

The service reads the table model only. Tree-model data becomes visible through a table view
(the path levels below the prefix become tags, the measurements become fields), for example for
devices root.factory.<line>.<machine>:

-- in the table model (CLI: -sql_dialect table)
CREATE DATABASE factory;
CREATE VIEW factory.machines (
  line STRING TAG,
  machine STRING TAG,
  temperature DOUBLE FIELD,
  running BOOLEAN FIELD
) AS root.factory.**;
iotdb=> \dv factory.*
         List of relations
 Schema  |   Name   | Type | Owner
---------+----------+------+-------
 factory | machines | view | root
(1 row)

iotdb=> SELECT line, avg(temperature) FROM factory.machines GROUP BY line ORDER BY line;
 line  |  avg
-------+-------
 line1 | 21.25
 line2 |  19.5
(2 rows)

The service does not create views itself; one view is needed per device pattern (also in the
module's README).

Design

Module and start. The new module external-service-impl/postgresql is a built-in external
service like REST and MQTT: a POSTGRESQL entry in BuiltinExternalServices, enabled by
enable_postgresql_service (default false). The jar-with-dependencies goes to lib/ of the
all and edge distributions and of the IT nodes. It has to be in lib/ and not in
ext/external_service, because the catalog tables (below) are registered through the class path
when the DataNode starts. The only third-party dependency is JSqlParser 5.4 (dual LGPL-2.1 /
Apache-2.0, used under Apache-2.0, added to LICENSE-binary). Netty comes from the DataNode.

Core changes (small):

  • BuiltinExternalServices.POSTGRESQL, plus IoTDBConfig/IoTDBDescriptor:
    enable_postgresql_service.
  • ExternalServiceClientSession: an IClientSession for clients that come in through an
    external service speaking another protocol. IClientSession's connection methods are
    package-private, so a plugin cannot subclass it itself.
  • iotdb-system.properties.template: the "PostgreSQL Service Configuration" section.
  • IT framework: typed DataNodeConfig setters for the service's keys.
  • Queries that read only information_schema tables run with the existing highest priority
    (Analysis.needSetHighestPriority, so far only for information_schema.queries). Catalog
    queries of clients join up to seven pg_* tables, and every sort of a join is estimated at a
    full sort buffer, so at small memory budgets they were refused with "There is not enough memory
    to execute current fragment instance", although their inputs are tiny. With the highest
    priority they use the existing memory fallback and are scheduled first. CI showed this on the
    Windows runner. QueryExecution now also sets the flag on the query context after
    distribution planning, because table-model analysis decides it after the context was created.

Protocol (server, protocol, session, execution): PostgreSQL protocol 3.0 on Netty:

  • connection start: SSLRequest, GSSENCRequest declined, CancelRequest, protocol negotiation;
  • simple and extended query protocol (Parse/Bind/Describe/Execute/Sync, portals), text and binary
    results;
  • SQL cursors (DECLARE/FETCH/MOVE/CLOSE, as psycopg's server-side cursors use them);
  • cancel requests (57014) and statement_timeout;
  • session commands handled by the service: SET/RESET/SHOW, BEGIN/COMMIT/ROLLBACK,
    DISCARD, DEALLOCATE;
  • streaming with back pressure: rows are sent block by block, and a client that stops reading
    for postgresql_service_write_timeout_in_ms loses its query and its connection;
  • a connection limit (53300), an idle timeout and a startup timeout;
  • engine errors mapped to SQLSTATEs (PgErrors).

Writes are refused with 25006. The reported server_version is 14.0, so clients take their
standard code paths.

Security:

  • On any address other than loopback the service accepts only TLS 1.2/1.3 connections with
    AEAD suites; a plain connection gets FATAL 28000 encryption required.
  • The certificate comes from key_store_path/key_store_pwd. Without a key store the service
    refuses to start off loopback and logs why.
  • Logins use PostgreSQL's cleartext password authentication. It is only ever offered inside
    TLS or on loopback, and the password is checked by SessionManager.login, so lockout and the
    other login rules apply unchanged. See "Differences" for why it isn't SCRAM.

Translation (sql): JSqlParser parses the statement. A check (ConstructCheck) allows only
constructs with an exact table-model equivalent, and a writer (PgToTableSql) prints table-model
SQL. Among the rules:

  • PostgreSQL output column names (count, ?column?, ...);
  • :: casts and typed literals;
  • ILIKE;
  • anchored regular expressions as LIKE/IN;
  • NULLIF, and ||/concat with PostgreSQL's NULL semantics;
  • typed NULLs;
  • date_trunc/extract/date_part and timestamp ± interval in the session time zone
    (DateTimeRules; day truncation is exact across DST changes, and units that would be wrong in
    the session zone are refused);
  • window functions without explicit frames;
  • scalar sub-queries;
  • the catalog functions clients call (format_type, pg_get_expr, obj_description,
    current_schemas, regclass/regtype casts, ...).

Catalog (catalog): the pg_catalog tables clients read (pg_namespace, pg_class,
pg_attribute, pg_type, pg_proc, pg_description, pg_database, pg_settings, ...) and
PostgreSQL's information_schema views. They are implemented as information_schema.pg_* /
pg_is_* tables through the existing InformationSchemaExtension and
AdditionalInformationSchemaProvider SPIs, so catalog queries, joins included, run in the normal
engine. They show only the databases and tables the user may read. OIDs are stable (CRC32 of the
name, at least 16384), and column categories (TAG/ATTRIBUTE/FIELD) appear as column comments.

Tests

  • Unit and golden tests (73): codec, startup on an in-memory channel, parameters of every type,
    value encoding, catalog rows, translation rules, and TranslatorGoldenTest. The golden test
    translates every recorded client statement and compares the result with
    corpus/expected/<client>/<n>.sql (or the expected 0A000).
  • Integration tests (org.apache.iotdb.postgresql.it, TableSimpleIT):
    • IoTDBPostgreSqlEngineIT runs every translated corpus statement on the engine and checks
      psql's catalog results, Npgsql's type-loading queries, date rules around both 2026 DST changes
      in Europe/Berlin, pgJDBC DatabaseMetaData, SQL cursors, cancel, statement_timeout and
      prepared statements with every parameter type;
    • IoTDBPostgreSqlClientsIT: pgJDBC simple/extended mode, metadata;
    • IoTDBPostgreSqlImportIT: 1,000,000 rows of every type through pgJDBC, compared with the
      session API, plus the write timeout;
    • IoTDBPostgreSqlSecurityIT: login, refusals, connection limit, read-only;
    • IoTDBPostgreSqlTlsIT: TLS-only off loopback, TLS 1.2/1.3, TLS 1.1 refused;
    • IoTDBPostgreSqlNoKeyStoreIT and IoTDBPostgreSqlPortInUseIT: the service fails cleanly and
      the DataNode keeps running.

Differences from the fork (each available as a separate PR if wanted)

The fork has a few things this PR deliberately leaves out, to keep the core changes small. If the
project is interested in any of them, I'm happy to send it as a separate PR:

  1. SCRAM-SHA-256 login. In the fork, built-in users log in with SCRAM-SHA-256, so the
    password never crosses the network, even inside TLS. That requires storing passwords as salted
    SCRAM verifiers ($scram-sha-256$<iterations>:<salt>$<StoredKey>:<ServerKey>, PBKDF2-SHA256)
    instead of the current unsalted SHA-256 hash. Existing hashes are converted when the ConfigNode
    starts or at the next login. That is an on-disk format change with no way back, so it needs its
    own discussion. Without it, this PR uses the password inside TLS.

  2. Plugin discovery instead of an enum entry. In the fork, optional services are found
    through ServiceLoader (META-INF/services/...IExternalService), and each plugin reads its own
    keys, so the core has no enable_postgresql_service and no POSTGRESQL entry. The fork also
    guards the start of discovered services against undeclared checked exceptions: Netty rethrows
    BindException undeclared, and the plugin catches it itself here too.

  3. Service status, settings and endpoints SPI. In the fork, external services report their
    status and endpoints to an embedding host (the fork has an embedded in-JVM mode). That isn't
    needed in IoTDB.

  4. Host user backends. The fork lets an embedding host plug in its own user management.
    PostgreSQL logins for such users use the password inside TLS as here, unless the backend
    supplies SCRAM verifiers.

  5. Configuration keys. The fork uses postgresql_service.* keys with ISO-8601 durations,
    because its plugins declare their settings through the SPI in item 3. This PR uses IoTDB's style
    (enable_postgresql_service, postgresql_service_*_in_ms).

  6. Uniform login refusals. In the fork, a wrong password and an unknown user get the same
    message (and a minimum response time), so logins don't reveal which users exist. This PR passes
    IoTDB's own login messages through ("The user ... does not exist"), with SQLSTATE 28P01 in
    both cases.

Open question: catalog tables while the service is off

The catalog tables (pg_class, pg_namespace, ... and the pg_is_* views, 29 tables) are
registered through InformationSchemaExtension when the jar is loaded. Because the jar ships in
lib/, they appear in every information_schema listing (SHOW TABLES FROM information_schema,
information_schema.tables/columns), even with enable_postgresql_service=false. That is
why this PR updates IoTDBDatabaseIT and IoTDBServicesIT.

Hiding them while the service is off would mean either making the registration depend on the
DataNode setting (InformationSchema lives in node-commons and is also loaded by the
ConfigNode, which doesn't know that setting), or filtering them out of the DataNode's
listings. Do you want them hidden while the service is disabled, or is listing them always
acceptable?

Help wanted: reviewing the golden files

external-service-impl/postgresql/src/test/resources/corpus/ contains SQL recorded from real
clients against PostgreSQL 14:

  • 21 statements from psql (\l, \dn, \dt, \d table, \d+);
  • 20 from pgJDBC (DatabaseMetaData calls as DBeaver makes them);
  • 14 from psycopg (named cursors, parameters);
  • 55 from Power BI Desktop 2.149 (Npgsql): the type loading after connecting, the navigator
    (information_schema.tables/columns, keys), an Import of a table and three DirectQuery
    visuals. All of them translate and run on the engine except pg_total_relation_size() (table
    size), which is refused; Power BI carries on without it.

corpus/expected/<client>/<n>.sql holds the expected translation of each statement, or the
expected 0A000 refusal. The translator itself generated these files
(-Dgolden.update=true), so for now they only prove that its output doesn't change, not that it
is right. IoTDBPostgreSqlEngineIT already proves that every translated statement runs on the
engine, and checks the rows for a few of them (psql's \dn and \d, pgJDBC metadata, the date
rules).

What we'd like reviewers to check, file by file: is the translation semantically the same as
the PostgreSQL statement, and is each refusal acceptable? A typical case is expected/psql/5.sql
(psql's \d table):

  • format_type(oid, NULL) becomes a CASE over the type OIDs;
  • pg_get_expr(...) becomes CAST(NULL AS STRING) (IoTDB has no column defaults);
  • relhasoids becomes FALSE.

Comments on individual files in this PR are very welcome. The Power BI files matter most,
especially the DirectQuery ones (expected/powerbi/50.sql to 54.sql): that is the SQL Power BI
sends for every visual. A recording of more DirectQuery visuals (date hierarchy, slicer, relative
date and Top N filters) will follow; if you use Power BI, recordings of your own reports are very
welcome too (corpus/README.md describes how they're made).


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors
    (external-service-impl/postgresql/README.md, configuration template).
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information (JSqlParser in
    LICENSE-binary).
  • added comments explaining the "why" and the intent of the code wherever would not be
    obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.

PostgreSQL clients (psql, pgJDBC/DBeaver, psycopg, Npgsql, Power BI, Excel)
can query the table model, read-only. New built-in external service
external-service-impl/postgresql, off by default (enable_postgresql_service):

- PostgreSQL protocol 3.0 on Netty: simple and extended query protocol,
  binary results, portals and SQL cursors, cancel, statement_timeout,
  streaming with a write timeout, connection limit.
- TLS 1.2/1.3 only off loopback (key_store_path); password login through
  SessionManager, offered only inside TLS or on loopback.
- Translation of PostgreSQL SQL into table-model SQL (JSqlParser); constructs
  without an exact equivalent are refused with SQLSTATE 0A000.
- pg_catalog and information_schema emulation as information_schema.pg_* and
  pg_is_* tables, filtered by the user's privileges.

Unit and golden tests over SQL recorded from psql, pgJDBC and psycopg, and
integration tests (TableSimpleIT).
Catalog queries of PostgreSQL clients (Npgsql's type loading joins seven pg_*
tables) were refused at small memory budgets with "There is not enough memory
to execute current fragment instance": every sort of a join is estimated at a
full sort buffer, however small the inputs are. Queries that read only
information_schema tables now get the existing highest priority (so far only
information_schema.queries had it), so they use the memory fallback and are
scheduled first. QueryExecution also sets the flag on the query context after
distribution planning, because table-model analysis decides it after the
context was created.
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.10955% with 1528 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.68%. Comparing base (dd450f7) to head (0b2fe71).

Files with missing lines Patch % Lines
...g/apache/iotdb/postgresql/session/SqlProtocol.java 5.19% 456 Missing ⚠️
...ache/iotdb/postgresql/session/SessionCommands.java 20.75% 126 Missing ⚠️
...e/iotdb/postgresql/server/PgConnectionHandler.java 50.23% 104 Missing ⚠️
...pache/iotdb/postgresql/execution/ResultCursor.java 0.00% 92 Missing ⚠️
...org/apache/iotdb/postgresql/PostgreSqlService.java 0.00% 82 Missing ⚠️
.../org/apache/iotdb/postgresql/sql/PgToTableSql.java 90.14% 82 Missing ⚠️
...pache/iotdb/postgresql/execution/RunningQuery.java 0.00% 76 Missing ⚠️
...che/iotdb/postgresql/protocol/BackendMessages.java 49.57% 59 Missing ⚠️
...he/iotdb/postgresql/session/StatementSplitter.java 0.00% 53 Missing ⚠️
...he/iotdb/postgresql/catalog/PgCatalogProvider.java 0.00% 45 Missing ⚠️
... and 29 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19350      +/-   ##
============================================
+ Coverage     49.44%   49.68%   +0.24%     
- Complexity      718     1454     +736     
============================================
  Files          5656     5703      +47     
  Lines        418222   423371    +5149     
  Branches      55296    56203     +907     
============================================
+ Hits         206769   210347    +3578     
- Misses       211453   213024    +1571     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- IoTDBServicesIT lists the POSTGRESQL built-in service.
- IoTDBDatabaseIT expects the information_schema tables that an
  InformationSchemaExtension (the PostgreSQL catalog) registers.
- The PostgreSQL ITs wait until the endpoint listens: the service starts
  after the DataNode reports itself ready (refused connections on Windows).
- Log checks wait for the expected line instead of reading the log once.
- The write-timeout IT starts its countdown when the first result bytes
  arrive, so slow machines that sort the result for a while do not finish
  the query before the stall.
@chrisdutz

chrisdutz commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

I am able to connect from Power BI using the built-in Postgres SQL driver:

Bildschirmfoto 2026-11-21 um 16 29 57 image

Power BI Desktop 2.149 recorded against PostgreSQL 14: connect (Npgsql
type loading), navigator, Import and three DirectQuery visuals, now in the
client corpus with the expected translations.

Power BI reads information_schema.character_sets on every connection; the
catalog now answers it as PostgreSQL 14 does for a UTF8 database with
collation C (UTF8 / UCS, default collation pg_catalog.ucs_basic), and
pg_collation lists ucs_basic. pg_total_relation_size stays refused.

extract.py accepts application names with spaces (Power BI sends
"Mashup Engine") and takes the capture date from the server log.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant